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GEOMETRIC ANALYSIS ON SMALL UNITARY 
REPRESENTATIONS OF GL(N, E) 

TOSHIYUKI KOBAYASHI, BENT 0RSTED, MICHAEL PEVZNER 

Abstract. The most degenerate unitary principal series repre- 
sentations mx,s (A £ 1, i £ Z/2Z) of G = GL(N, R) attain the 
minimum of the Gelfand-Kirillov dimension among all irreducible 
unitary representations of G. This article gives an explicit for- 
mula of the irreducible decomposition of the restriction %i\ t $\H 
(branching law) with respect to all symmetric pairs (G,H). For 
N = 2n with n > 2, the restriction TTi\ t s\ii remains irreducible 
for H = Sp(n,M) if A ^ and splits into two irreducible repre- 
sentations if A = 0. The branching law of the restriction ~Ki\^\ii 
is purely discrete for H = GL(n,C), consists only of continuous 
spectrum for H = GL(p, R) x GL(q, R) (p + q = N), and contains 
both discrete and continuous spectra for H = 0(p,q) (p > q > 1). 
Our emphasis is laid on geometric analysis, which arises from the 
restriction of 'small representations' to various subgroups. 

Key words and phrases: small representation, branching law, 
symmetric pair, reductive group, phase space representation, sym- 
plectic group, degenerate principal series representations. 



1. Introduction 

The subject of our study is geometric analysis on 'small representa- 
tions' of GL(N,M.) through branching problems to non-compact sub- 
groups. 

Here, by a branching problem, we mean a general question on the 
understanding how irreducible representations of a group decompose 
when restricted to a subgroup. A classic example is studying the ir- 
reducible decomposition of the tensor product of two representations. 
Branching problems are one of the most basic problems in represen- 
tation theory, however, it is hard in general to find explicit branching 
laws for unitary representations of non-compact reductive groups. For 
reductive symmetric spaces G/H, the multiplicities in the Plancherel 
formula of L 2 (G/H) are finite [TJ [5], whereas the multiplicities in the 
branching laws for the restriction G \. H are often infinite even when 
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(G, H) are symmetric pairs (see e.g. p2] for recent developments and 
open problems in this area). 

Our standing point is that 1 small representations 7 of a group should 
have 'large symmetries' in the representation spaces, as was advocated 
by one of the authors from the perspectives in global analysis |18j . 
In particular, considering the restrictions of 'small representations' to 
reasonable subgroups, we expect that their breaking symmetries should 
have still fairly large symmetries, for which geometric analysis would 
deserve finer study. 

Then, what are 'small representations'? For this, the Gelfand- 
Kirillov dimension serves as a coarse measure of the 'size' of infinite 
dimensional representations. We recall that for an irreducible unitary 
representation tt of a real reductive Lie group G the Gelfand-Kirillov 
dimension DIM (71*) takes the value in the set of half the dimensions of 
nilpotent orbits in the Lie algebra g. We may think of tt as one of the 
'smallest' infinite dimensional representations of G, if DLM(7r) equals 
n(G), half the dimension of the minimal nilpotent orbit. 

For the metaplectic group G = Mp(m,M.), the connected two- fold 
covering group of the symplectic group Sp(m, K) of rank m, the Gelfand- 
Kirillov dimension attains its minimum n(G) — m at the Segal-Shale- 
Weil representation. For the indefinite orthogonal group G = 0(p,q) 
(p,q > 3), there exists tt such that DIM(7r) = n{G) (= p + q — 3) if 
and only if p + q is even according to an algebraic result of Howe and 
Vogan. See e.g. a survey paper [12] for the algebraic theory of 'minimal 
representations', and [HI [TU [191 [20] f° r their analytic aspects. 

In general, a real reductive Lie group G admits at most finitely many 
irreducible unitary representations tt with DIM(7r) = n(G) if the com- 
plexified Lie algebra gc does not contain a simple factor of type A (see 
[12]). In contrast, for G = GL(N,M), there exist infinitely many irre- 
ducible unitary representations tt with DLM(7r) = n(G) (= N — 1). For 
example, the unitarily induced representations 

(1-1) C* sI CW) 

from a unitary character Xi\,s of a maximal parabolic subgroup 
(1.2) Pn ■= (GL(1,M) x GL(N — 1, R)) 



tN-l 



arc 



such representations with parameter A G K and S G Z/2Z. 

In this paper, we find the irreducible decomposition of these 'small 
representations' nf x L ^ N ' R ^ with respect to all symmetric pairs. 

We recall that a pair of Lie groups (G, H) is said to be a symmetric 
pair if there exists an involutive automorphism a of G such that H is an 
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open subgroup of G a := {g G G : og = g}. According to M. Berger's 
classification [4], the following subgroups H = K, Gj (1 < j < 4) and 
G exhaust all symmetric pairs (G,H) for G = GL(N,R) up to local 
isomorphisms and the center of G: 



K 

G 1 
G 2 
G 3 
G 4 



O(N) 
Sp{n, R) 
GL(n,C) 

GL(p,R) x GL(q,'. 
0(p,q) 



(maximal compact subgroup), 
(N = 2n), 
(N = 2n), 
(N = p + q), 
(N = p + q). 



GL(N,\ 



It turns out that the branching laws for the restrictions of TT iX 5 
with respect to these subgroups behave nicely in all the cases, and 
in particular, the multiplicities of irreducible representations in the 
branching laws are uniformly bounded. 

To be more specific, the restriction of 7rf x L j N ' R ^ to K splits discretely 
into the space of spherical harmonics on R N , and the resulting .fT-type 
formula is multiplicity-free and so called of ladder type. For the non- 
compact subgroups Gj (1 < j < 4), we prove the following irreducible 
decompositions in Theorems 18.11 19.11 110.11 and 111.11 

Theorem 1.1. For A G R and 5 G Z/2Z, the irriducible unitary rep- 
decomposes when restricted to symmetric pairs as 



resentation tt- 



GL(N,l 



X,5 



follows: 

1) GL(2n, 



TT 



GL(2n,\ 
X,5 



G 



I Sp(n,R), (n > 2): 
Irreducible 

Sp{n,\ 



TT, 



0,5 



TT, 



Sp{n,R) 
0,5 



(A t^O), 
(A = 0). 



2) GL(2n, R) I GL(n, C) : 



7i 



GL(2n,\ 
i.X,5 



^ — \© 

- E ' 

m£2Z+6 

3) GL(p + q, R) I GL(p, R) x GL(q, 1 



G 2 



GL(n,( 
iX,m 



TT. 



GL(p+q,\ 
i\,5 



G 3 



<5'eZ/2Z 



4) GL(p + q,R)lO(p,q): 



GL{p,l 
^iX'J' 



GL(g,R) „, 
i(\—\'),5—5' ■ 



TT: 



GL{p+q,l 



iX,5 



Gi 



71 



0(p,q) 
+ -V 



vdA 6 + {p,q) 



e E 

v&AUq,p) 



TT°_^ q) ® 2 



^} P 5 q) dv. 

1V.0 
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Here, each summand in the right-hand side stands for (pairwise in- 
equivalent) irreducible representations of the corresponding subgroups 
which will be defined explicitly in Sections El [9j [10] and [TTJ 

As indicated above, we see that the representation nf^} 2 ™'^ remains 
generically irreducible when restricted to the subgroup Gi = Sp(n, M.) 
and splits into a direct sum of two irreducible subrepresentations for 
A = and n > 1. The case n — 1 is well known (cf. [3]): the group 
is isomorphic to SX(2,R), and 7Tj>,($ are irreducible except for 
(A, S) = (0,1), while 7r 0) i splits into the direct sum of two irreducible 
unitary representations i.e. the (classical) Hardy space and its dual. 

The representation 7r£^ 2n,It ^ is discretely decomposable in the sense 
of [16] when restricted to the subgroup G2 = GL(n,C). In other 
words, the non-compact group G2 behaves in the representation space 
of ^x5 2n ^ as ^ ^ were a compact subgroup. In contrast, the re- 
striction of vr^ L / p+9 ' M ' ) to another subgroup G 3 = GL(p,M) x GL(q,~R.) 
decomposes without discrete spectrum, while both discrete and contin- 
uous spectra appear for the restriction of 7r^5 9 ' K ^ to G4 = 0(p, q) if 
p, q > 1 and (p, q) ^ (1,1)- Finally, in Theorem 112. II we give an irre- 
ducible decomposition of the tensor product of the Segal-Shale-Weil 
representation with its dual, giving another example of explicit branch- 
ing laws of small representations with respect to symmetric pairs. 

We have stated Theorem 11.11 from representation theoretic view- 
point. However, our emphasis is not only on results of this nature 
but also on geometric analysis of concrete models via branching laws 
of small representations, which we find surprisingly rich in its interac- 
tion with various domains of classical analysis and their new aspects. 
It includes the theory of Hilbert-space valued Hardy spaces (Section 
2), the Weyl operator calculus (Section 3), representation theory of Ja- 
cobi and Heisenberg groups, the Segal-Shale- Weil representation of the 
metaplectic group (Section 4), (complex) spherical harmonics (Section 
5), the K-Bessel functions (Section [7]), and global analysis on space 
forms of indefinite- Riemannian manifolds (Section [TT]) . 

Further, we introduce a non-standard L 2 -model for the degenerate 
principal series representations of Sp(n, M) where the Knapp-Stein in- 
tertwining operator becomes an algebraic operator (Theorem 16.11) . In 
this model the minimal .ff -types are given in terms of Bessel functions 
(Proposition ^. ip . The two irreducible components 71"^ at A = in The- 
orem [TJTJ 1) will be presented in three ways, that is, in terms of Hardy 
spaces based on the Weyl operator calculus as giving the P-module 
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structure, complex spherical harmonics as giving the i^-module struc- 
ture, and the eigenspaces of the Knapp-Stein intertwining operators 
(see Theorem 18.31) . 

The authors are grateful to an anonymous referee for bringing the 
papers of Barbasch [2J and Farmer [TO] to our attention. 



Notation: N = {0,1,2,...}, N+ = {1, 2, 3, . . . }, R± = {p e K : 

±p > 0}, M x = E \ {0}, and C x = C \ {0}. 



Let W be a (separable) Hilbert space. Then, we can define the 
Bochner integrals of weakly measurable functions on K with values in 
W. For a measurable set E in R, we denote by L 2 (E, W) the Hilbert 
space consisting of H^-valued square integrable functions on E. Clearly, 
it is a closed subspace of L 2 (M, W). 

Suppose F is a H^-valued function defined on an open subset in C. 
We say F is holomorphic if the scalar product (F, w)w is a holomorphic 
function for any w G W. 

Let n + be the upper half plane {z = it + iu 6 C : m = Imz > 0}. 
Then, the W-valued Hardy space is defined as 



U%{W) := {F : n + -»■ W : F is holomorphic and \ \F\\ H 2 +{W) < oo}, 
where the norm ||-F||^2 (W) is given by 



\u>0 Jr / 

Similarly, ^{^(W) is defined by replacing n + with the lower half 
plane n_. Notice that %\{W) is the classical Hardy space, if W — C. 

Next, we define the W- valued Fourier transform T as 



Here, the Bochner integral converges for / e (L 1 fl L 2 )(M, W) with 
obvious notation. Then, F extends to the Hilbert space L 2 (IR, W) as a 
unitary isomorphism. 

Example 2.1. Suppose W = L 2 (R. k ) for some k. Then, we have a 
natural unitary isomorphism L 2 (R,W) ~ L 2 (R k+1 ). Via this isomor- 
phism, the L 2 (M. k ) -valued Fourier transform F is identified with the 



2. Hilbert space valued Hardy space 



(2.1) 
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partial Fourier transform T t with respect to the first variable t as fol- 
lows: 

L 2 (R,L 2 (R k )) 3- L 2 (R,L 2 (R k )) 

(2.2) \l \l 
L 2 (R k+1 ) ^ L 2 (R k+1 ). 

Ft 

As in the case of the classical theory on the (scalar-valued) Hardy 
space %\ = "H+(C), we can characterize H±(W) by means of the 
Fourier transform: 

Lemma 2.2. Let W be a separable Hilbert space, and1-l±(W) the W- 
valued Hardy spaces (see ( 12. II) ). 

1) For F G H^W), the boundary value 

F(t ± iO) := \imF(t ± iu) 

exists as a weak limit in the Hilbert space L 2 {R, W), and defines 
an isometric embedding: 

(2.3) H 2 ±(W) ^ L 2 (R, W). 

From now, we regard l-i\{W) as a closed subspace of L 2 (R, W). 

2) The W -valued Fourier transform J 7 induces the unitary isomor- 
phism: 

7 : T-L 2 ±(W) ^L 2 (R±,W). 

3) L 2 (R, W) = U 2 + {W) © U 2 _{W) (direct sum). 

4) If a function F G U%(W) satisfies F(t + iO) = F(-t + iO) then 
F = 0. 

Proof. The idea is to reduce the general case to the classical one by 
using a uniform estimate on norms as the imaginary part u tends to 
zero. 

Let {ej} be an orthonormal basis of W. Suppose F G ?i+(W). Then 
we have 

W F \\nl(w) = SU P / WFit + iujWwdt 

(2.4) = sup V /,(«), 

M>0 

3 

where we set 

J j( u ) '■= / \( F (t + iu),e j ) w \ 2 dt. 
Jr 
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Then, it follows from ( 12^41) that for any j sup M>0 Ij(vi) < oo and there- 
fore 

Fj(z) := {Fj{z), e 3 ) w , [z = t + iu G n+) 

belongs to the (scalar- valued) Hardy space l-i 2 + . By the classical Paley- 
Wiener theorem for the (scalar- valued) Hardy space we have: 

(2.5) Fj(t + iO) := \imFj(t + iu) (weak limit in L 2 (M)), 

(2.6) FFj(t + i0) G L 2 (R+), 

(2.7) (JFi^t + = e- 2 ™ p (TFj(t + iO))(p) for u > 0, 

(2.8) -^j( M ) is a monotonely decreasing function of u > 0, 

(2.9) lim^-H = llF^t + m)!!^ = + i0)||| 2(R) . 

The formula (12. 7p shows (12. 8p , which is crucial in the uniform estimate 
as below. In fact by (12.81) we can exchange sup u>0 and in (12.41) . 
Thus, we get 

\\ f \\hi ( w) = E^ 7 iN = E + W 

i i 

Hence we can define an element of L 2 (M, W) as the following weak 
limit: 

j 

Equivalently, F(t + iO) is the weak limit of F(t + iu) in L 2 (IR, W) as 
u — > 0. Further, (12. 6p implies supp FF{t + iO) C IR+ because 

+ iO) = FFA 1 + *°) e i ( weak limit )- 

3 

In summary we have shown that F(t + iO) G L 2 (IR, H 7 ), JFF(t + zO) G 
L 2 (M+,H/), and 

\\F\\h\{w) = \\F(t + iO)\\v<BL,w) = ||^(* + ^)||i, 2 ( R+) w) 

for any F G %+(W). Thus, we have proved that the map 

F :H 2 + (W) ^L 2 (R+,W) 

is well-defined and isometric. 

Conversely, the opposite inclusion F~ 1 (L 2 (M. + , W)) C l-i 2 + (W) is 
proved in a similar way. Hence the statements 1), 2) and 3) follow. 

The last statement is now immediate from 2) because FF(t+iO)(p) = 
FF(-t + iO)(-p). □ 



8 TOSHIYUKI KOBAYASHI, BENT 0RSTED, MICHAEL PEVZNER 



3. Weyl Operator Calculus 

In this section, based on the well-known construction of the Schrodinger 
representation and the Segal-Shale-Weil representation, we introduce 
the action of the outer automorphisms of the Heisenberg group on the 
Weyl operator calculus (see (13.111) . (I3.13p . and (I3.14p ). and discuss 
carefully its basic properties, see Proposition 13.21 and Lemma 13.41 In 
particular, the results of this section will be used in analyzing of the 
'small representation' vr^^ 2 ™' 1 ^, when restricted to a certain maximal 
parabolic subgroup of Sp(n,M.), see e.g. the identity (I4.12p . 

Let lR 2m be the 2m-dimensional Euclidean vector space endowed with 
the standard symplectic form 

(3.1) u(X, Y) = co((x, o, (y, v)) ■= (£, y) - (x, v)- 

The choice of this non-degenerate closed 2-form gives a standard 
realization of the symplectic group Sp(m, R) and the Heisenberg group 
H 2m+1 . Namely, 

Sp(m, R) := {T G GL(2m, R) : u(TX, TY) = u(X, Y)} 

and 

equipped with the product 

g.g'= (s, A) ■ (*', A') := (s + s' + ±u(A, A'),A + A'). 

Accordingly, the Heisenberg Lie algebra (j 2m+1 is then defined by 

[(s,X),(t,Y)} = (u(X,Y),0). 

Finally we denote by Z the center {(s, 0) : s G M} of H 2m+1 . 
The Heisenberg group H 2m+1 admits a unitary representation, de- 
noted by on the configuration space L 2 (M. m ) by the formula 

(3.2) 0(g)<p(x) = e 2ni{s+{x ' a) - 1 * {a ' a)) V {x-a), g = (s,a,a). 

This representation, referred to as the Schrodinger representation, is 
irreducible and unitary [23]. The symplectic group, or more precisely 
its double covering, also acts on the same Hilbert space L 2 (M. m ). 

In order to track the effect of Aut(iJ 2m+1 ), we recall briefly its 
construction. The group Sp(m, R) acts by automorphisms of H 2m+1 
preserving the center Z pointwise. Composing i? with such automor- 
phisms T G Sp(m, R) one gets a new representation i9 o T of H 2m+1 
on L 2 (IR m ). Notice that these representations have the same central 
character, namely i?oT(s,0,0) = e 2ms id = $(s, 0,0). According to 
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the Stone-von Neumann theorem (see Fact 13.31 below) the representa- 
tions ■& and doT are equivalent as irreducible unitary representations of 
H 2m+1 . Thus, there exists a unitary operator Met(T) acting on L 2 (IR m ) 
in such a way that 

(3.3) (0 o T) (g) = Met{T)${g)Met{T)-\ g G H 2m+1 . 

Because $ is irreducible, Met is defined up to a scalar and gives rise 
to a projective unitary representation of Sp(m, R). It is known that 
this scalar factor may be chosen in one and only one way, up to a sign, 
so that Met becomes a double- valued representation of Sp(m, R). The 
resulting unitary representation of the metaplectic group, that we keep 
denoting Met, is referred to as the Segal-Shale-Weil representation and 
it is a lowest weight module with respect to a fixed Borel subalgebra. 
Notice that choosing the opposite sign of the scalar factor in the defi- 
nition of Met one gets a highest weight module which is isomorphic to 
the contragredient representation Met v . 

The unitary representation Met splits into two irreducible and in- 
equivalent subrepresentations Meto and Meti according to the decom- 
position of the Hilbert space L 2 (R m ) = L 2 (R m ) even © L 2 (R m ) odd . 

The Weyl quantization, or the Weyl operator calculus, is a way to 
associate to a function & (x, £) the operator Op((5) on L 2 (IR m ) defined 
by the equation 

(3.4) (0p(6)u)(z) = f & v) e 2ni{x - y '^ u(y) dydrj. 

Such a linear operator sets up an isometry 

(3.5) Op : L 2 {R 2m ) ^ HS(L 2 (R m ), L 2 (IR m )). 

from the phase space L 2 (]R m x IR m ) onto the Hilbert space consisting 
of all Hilbert-Schmidt operators on the configuration space L 2 (IR m ) . 
Introducing the symplectic Fourier transformation J-" symp by: 

(3.6) (J- symp &){X) := / &(Y) e - 2 ^ x ' ¥ > dY , 

JR m xM. m 

one may give another, fully equivalent, definition of the Weyl operator 
by means of the equation 

(3.7) Op(©)= f (F symp e)(Y)$(0,Y)dY, 

JR 2m 

where the right-hand side is a Bochner operator-valued integral. 
The Heisenberg group H 2m+1 acts on R 2m ~ H 2m+1 /Z, by 

R 2m -> R 2m , X^X + A for g = (s, A), 
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and consequently it acts on the phase space L 2 (R 2m ) by left transla- 
tions. The symplectic group Sp(m, R) also acts on the same Hilbert 
space L 2 (R 2m ) by left translations. (This representation is reducible. 
See Section [12] for its irreducible decomposition.) In fact, both repre- 
sentations come from an action on L 2 (R 2m ) of the semidirect product 
group G J := Sp(m, R) x H 2m+1 which is referred to as the Jacobi group. 

Let us recall some classical facts in a way that we shall use them in 
the sequel: 

Fact 3.1. 

1) The representations $ and Met form a unitary representation 
of the double covering Mp(m, R) x H 2m+1 of G J on the con- 
figuration space L 2 (R m ). This action induces a representation 
of the Jacobi group G J on the Hilbert space of HUbert-Schmidt 
operators HS(L 2 (R m ), L 2 (R m )) by conjugations. 

2) The Weyl quantization map Op intertwines the action of G J on 
L 2 (R 2m ) with the representation Met x •d on the Hilbert space 
HS(L 2 (R m ),L 2 (M m )) defined in 2). Namely, 

(3.8) d(g) 0p(6) tfGT 1 ) = 0p(6 o g' 1 ), g e H 2m+1 . 

(3.9) Met(g)Op(&)Met-\g) = Op(e o g- 1 ), g e Sp(m,R). 

3) Any unitary operator satisfying A3.SA) and A3. 9\) is a scalar mul- 
tiple of the Weyl quantization map Op. 

Proof. Most of these statements may be found in the literature (e.g. 
[TTJ Chapter 2] for the second statement), but we give a brief expla- 
nation of some of them for the convenience of the reader. Namely, 
the first statement follows from (13.31) . Consequently, the semi-direct 
product Mp(m, R) x H 2m+1 also acts by conjugations on the space 
HS(L 2 (R m ),L 2 (R m )), and this action is well defined for the Jacobi 
group G J = Sp(m, R) x H 2m+1 because the kernel of the metaplectic 
cover Mp(m,R) -»■ Sp(m,R) acts trivially on HS(L 2 (M m ), L 2 (R m )). 

The third statement follows from the fact that L 2 (R 2m ) is already 
irreducible by the codimension one subgroup Sp(m, R) x R 2m of G J . In- 
deed, any translation-invariant closed subspace of L 2 (R 2m ) is a Wiener 
space, i.e. the pre- image by the Fourier transform of L 2 (E) for some 
measurable set E in M. 2m . On the other hand, the symplectic group 
acts ergodically on R 2m , in the sense that the only Sp(m, R)-invariant 
measurable subsets of lR 2m are either null or conull with respect to the 
Lebesgue measure. Hence, the whole group Sp(m, R) x R 2m+1 acts 
irreducibly on L 2 (R 2m ). □ 
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Now we consider the 'twist' of the metaplectic representation by 
automorphisms of the Heisenberg group. 

The group of automorphisms of the Heisenberg group H 2m+1 , to be 
denoted by Aut(if 2m+1 ) , is generated by 

- symplectic maps : (s, A) i-> (s,T(A)), where T G Sp(m, E); 

- inner automorphisms (s, A) \-t Iu B )(s,A) := (t , B) (s , A) (t , B)~ ] 
= {s - u(A, B),A), where (t, B) G H 2m+1 ; 

- dilations (s, A) !->■ d(r)(s,A) := (r 2 s,rA), where r > 0; 

- inversion: (s, A) i— >■ z(s, A) := (— s, a, a), where A = (a, a). 

In the sequel we shall pay a particular attention to the reseating map 
t p which is defined for every p ^ by 

(3.10) r p : H 2m+1 H 2m+ \ (s,a,a)^ (|s,a,|or). 

Here we have adopted the parametrization of r p in a way that it fits 
well into Lemma H~2l We note that (t_ 4 ) 2 = id and r 4 = id. 

The whole group Aut(H 2m+1 ) of automorphisms is generated by G J 
and {r p : p G 1R X }. We denote by Aut(iJ 2m+1 ) the identity component 
of Aut(# 2m+1 ). Then we have 

Aut(# 2m+1 ) = {l,r_ 4 } • Aut(# 2m+1 ) . 



For any given automorphism r G Aut(if 2m+1 ), we denote by r the 
induced linear operator on H 2m+1 /Z ~ IR 2m and by 7t(t) its pull-back 
7r(r)/ := / o (r) _1 . We notice that tt(t) is a unitary operator on 
L 2 (M 2m ) if r G G J . 

Further, we define the r- twist Op T of the Weyl quantization map Op 

by 

(3.11) Op T := Opo7r(r). 

In particular, it follows from (13 .4p and (I3.10p that 
(3.12) 

(Op T ,(6)Ti)(aO = / & (Z±1L, -t) e 2 ^-y^u(y) dyd V . 

Similarly, we define the r-twist d T of the Schrodinger representation 
tfby 

(3.13) $ t -=$°t-\ 

Finally, we define the r-twist Met T of the Segal-Shale- Weil representa- 
tion Met. For this, we begin with the identity component Aut(if 2m+1 ) G . 
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We set 

(3.14) Met T := A' 1 o Met o A, where 

(Met(r), for r G Sp(m, M), 
0(r), forrei? 2m+1 , 
Id, for r = d(r). 

It follows from FactOl) that Met r is well-defined for r G Aut(H 2m+1 ) . 
For the connected component containing r_4, we set 

(3.15) Met T := (Met T V 

for r = t_ 4 t', t' G Aut(F 2m+1 ) . 

Thereby, Met r is a unitary representation of Mp(m,M.) on L 2 (IR m ) 
characterized for every T G Sp(m, M) by 

M et T (T)$ T (g) M et T (T)~^ = A(T(g)). 

Hence, the group Aut(_£f 2m+1 ) acts on L 2 (IR 2m ) in such a way that 
the following proposition holds. 

Proposition 3.2. 

1) The r-twisted Weyl calculus is covariant with respect to the Ja- 
cobi group: 

(3.16) Mg) 0p T (6) M^ 1 ) = OPr(© o g- 1 ), g G iJ 2m+1 , 

(3.17) Met T (( 7 )Op T (6)Met; 1 (( 7 ) = Op T (6 o g~ l ), g e Sp(m,R). 

2) For any r G Aut(H 2m+1 ) the representation Met T is equivalent 
either to Met or to its contragredient Mef . 

The special case of the r- twist, namely, the r- twist associated with 
the rescaling map r p (13 . 10 j) deserves our attention for at least the fol- 
lowing two reasons. First, the parameter | has a concrete physical 
meaning - this is the inverse of the Planck constant h (see pT, The- 
orem 4.57], where a slightly different notation was used. Namely, the 
Schrodinger representations that we denote by d Tp correspond therein 
to ph with h — -). Secondly, dilations do not preserve the center Z of 
the Heisenberg while the symplectic automorphisms of H 2m+1 do. More 
precisely, the whole Jacobi group G J fixes Z pointwise. The last ob- 
servation together with the Stone - von Neumann theorem (see below) 
shows that the action of Aut(# 2m+1 ) / G J ~ {r p : p G M x }(~ M x ) is 
sufficient in order to obtain all infinite dimensional irreducible unitary 
representations of the Heisenberg group. 

We set 



(3.18) 



U p ■ U T p 1 
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to which we refer as the Schrddinger representations with central char- 
acter p. 

Fact 3.3 (Stone-von Neumann Theorem, [T3j [23] ) - The representa- 
tions dp constitute a family of irreducible pairwise inequivalent unitary 
representations with real parameter p. Any infinite dimensional irre- 
ducible unitary representation of H 2m+1 is uniquely determined by its 
central character and thus equivalent to one of the d p 's. 

To end this section, we give yet another algebraic property of the 
Weyl operator calculus. We shall see in Lemma H~5l that the irreducible 
decomposition of 7r^^ 2n,R \ when restricted to a maximal parabolic sub- 
group of Sp(n, R), is based on an involution of the phase space coming 
from the parity preserving involution on the configuration space. 

Consider on L 2 (IR m ) an involution defined by u(x) := u(— x) and 
induce through the map Op rp : L 2 (R 2m ) HS(L 2 (M m ), L 2 (R m )) two 
involutions on L 2 (IR 2m ), denoted by & H- and & h-> (5^p, by the 
following identities: 



Then 'p© and &p are characterized by their partial Fourier transforms 
defined by 



(3.19) 
(3.20) 



Op Tp (tp©)(n) = 0p Tp (6)(«), 
Op Tp (e^)(u) = (Op Tp (6) (u)J. 




Lemma 3.4. 




Proof. By (I3.12p the first equality (13 . 1 9[) amounts to 




The right-hand side equals 
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This equality holds for all u G L 2 (M. m ), and therefore, 



Namely, 



Thus the first statement follows and the second may be proved in the 
same way. □ 



4. Restriction of 7^,5 to a maximal parabolic subgroup 

Let n — m + 1. Consider the space of homogeneous functions 
(4.1) 

V% := {/ e C™(R 2n \ {0}) : f(r-) = {sgnrf\r\- n ^f(-),r G M x }, 

for 5 = 0, 1 and /x G C. It may be seen as the space of even or 
odd smooth functions on the unit sphere S* 2 ™ -1 according to 5 = or 
1, since homogeneous functions are determined by their restriction to 
S* 2 ™ -1 . Let V^g denote its completion with respect to the L 2 -norm over 
iS 2 " -1 . Likewise, by restricting to the hyperplane defined by the first 
coordinate to be 1, we can identify the space V^s with the Hilbert space 
L 2 (IR 2n_1 ) up to a scalar multiple on the inner product. 

The normalized degenerate principal series representations vr^^ 2n ' R ^ 
induced from the character Xn,s °f a maximal parabolic subgroup P2 n 
of GL(2n,M.) corresponding to the partition 2n = 1 + (2n — 1) may 
be realized on these functional spaces. The realization of the same 
representation on V^g will be referred to as the K-picture, and on 
L 2 (R 2n ~ 1 ) as the N -picture. 

In addition to these standard models of ir^ ( ' 2n ' M '\ we shall use an- 
other model L 2 (M,HS(L 2 (M m ),L 2 (M m ))), which we call the operator 
calculus model. It gives a strong machinery for investigating the restric- 
tion to the maximal parabolic subgroup of Sp(n,M.) (see (14. 3 p below). 

Let us denote by 



F t (f)(p,X)= [ f(t,X)e- 2 ^dt } 



the partial Fourier transform of f(t,X) G L 2 (]R 1+2m ) with respect to 
the first variable. Applying the direct integral of the operators Op T 
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and using (12. 2D . we obtain the unitary isomorphisms 



(4.2) 



V, 



1 1, 5 



fOp Tp dp 



-2/Trpl+2m\ 



L (R, L (R )) ~ L (R, L (R )) 



L 2 (M,HS(L 2 (M m ),L 2 (M m ))). 



According to situations we shall use following geometric models for 
the induced representations: 



standard 


model 


V^s — L 2 (S 2n ~ l )s 


i^-picture 


'\^rcstrict 




V$ = {f e C°°(R 2n 


\ {0}) : f(rX) = \r\-»- n (sgnr) s f(X),r e R x } 


^//restrict 




L 2 (H 2m+1 ) = L 2 (R,L 2 (R 2m )) 


iV-picture 



L 2 (R,L 2 (R 2m )) ~ L 2 (M 2m+1 ) 





non-standard mode 



operator calculus model 



Figure 4.1. 

The group G\ = Sp(n,R)(= Sp{m + 1,R)) acts by linear symplec- 
tomorphisms on IR 2n and thus it also acts on the real projective space 
p 2m+i M _ Fix a point in F 2m+i M and denote by p its stabilizer in G 1 . 

This is a maximal parabolic subgroup of G\ with Langlands decompo- 
sition 



(4.3) 



P = MAN 



Sp(m,R)) x H 



2m+l 



Let 0i = n + m+ a + nbe the Gelfand-Naimark decomposition for 
the Lie algebra 0i = Lie{G\). 
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(4.4) (s,x,C)^ 



We identify the standard Heisenberg Lie group H 2m+1 with the sub- 
group N = exp n through the following Lie groups isomorphism: 

/ 1 \ 

X I m 

2s *f 1 - l x 

\i l m ) 

Thus, in the coordinates (t,x,£) G H 1+2m , the restriction map V^° s — > 
L 2 (H 2m+1 ) is given by 

(4.5) f^f{l,2t,x,0- 

The action of G\ on P 2n_1 ]R is transitive, and all such isotropy sub- 
groups are conjugate to each other. Therefore, we may assume that 
P = Sp(n, E) flP2n- Then, the natural inclusion Sp(n, E) C GL(2n, E) 
induces the following isomorphisms 

Sp(n, E) /P ^> GL(2n, E) /P 2n ~ P^E. 

Hence, the (normalized) induced representation = 7r^ n ' M ' ) : = 
Indp P • > x / u,<5 can i c f- Section [8]) also be realized on the Hilbert space 
V^s- Therefore, ir^s is equivalent to the restriction of 7r ^( 2n,IR ) with 
respect to Sp(n, E). Notice that ir^s is unitary for // = zA, A G E. 

It is noteworthy that the unipotent radical iV of P is the Heisenberg 
group H 2n ~ l which is not abelian if n > 2, although the unipotent 
radical of P2„ clearly is. Notice also that the automorphism group 
Aut(P 2 " -1 ) contains P/{±1} as a subgroup of index 2. 

Denote by M Q ~ S£>(m,E) the identity component of M ~ 0(1) x 
Sp(m, E). The subgroup M G x iV is isomorphic to the Jacobi group G J 
introduced in Section [31 

We have then the following inclusive relations for subgroups of sym- 
plectomorphisms: 

Gi D MAN D G J = M Q N D N. 

Symplectic group Jacobi group Heisenberg group 

Our strategy of analyzing the representations 7r iA 5 of Gi (see The- 
orem I8.3p will be based on their restrictions to these subgroups (see 
Lemmas 14.11 and 14.51) . 

We recall from (13.1 8p that t? p is the Schrodinger representation of the 
Heisenberg group H 2m+1 with central character p. While the abstract 
Plancherel formula for the group N ~ H 2m+1 : 

L 2 (N) = [ # p ®# w p dp, 
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underlines the decomposition with respect to left and right regular ac- 
tions of the group N, we shall consider the decomposition of this space 
with respect to the restriction of the principal series representation ir iXt s 
to the Jacobi group G J = Sp(m, R) x H 2m+1 (see Lemma [4. II) . 

Let us examine how the restriction TTix t s\c J defined on the Hilbert 
space V iX ,6 on the left-hand side of g^D is transferred to L 2 (R, L 2 (R 2m )) 
via the partial Fourier transform 

The restriction 71^,5 |jv coincides with the left regular representation 
of N on L 2 {R l+2m ) given by 

(4.6) ir iX>s (g)f(t, X) = f{t - s - ~cj(A, X),X - A) 

= f(t-s + -((Z,a)-(x,a)),x-a,£-a), 

for f(t,X) G L 2 (R 1+2m ) and g = (s,A) = (s,a,a) G U 2m+1 . 

Taking the partial Fourier transform T% of (14. 6p . we get 
(4.7) 

[Tt M9) f))(p,x, = c -*Kp(-i(«.->-(».«») (F t f)(p,x - a, e - a). 
Now, for each p 6 M, we define a representation w p of N on L 2 (M. 2m ) 

by 

(4.8) w P (g)h{x, := e -2«P(«-4 («;><*>-(*,<*») _ a> £ _ a ) j 

for (7 = (s, a, a) G iV and ft G L 2 (IR 2m ). Then, -07 p is a unitary repre- 
sentation of iV for any p, and the formula (14.71) may be written as: 

(4.9) (F t n iX ,s(g)f)(p,x,0 = tu p (g)(F t f)(p,x,0, 

for g G N. Here, we let m p (g) act on JF t f seen as a function of (x,£). 

For each p G K, we can extend the representation tu^ of N to a 
unitary representation of the Jacobi group G J by letting M Q act on 
L 2 (R 2m ) by 

zu p (g)h(x, £) = h(y, 77), with (y, 77) = -1 (a:, (),36M„ - Sp(m, E). 

Then, clearly the identity (14.91) holds also for g G M a . Thus, we have 
proved the following decomposition formula: 

Lemma 4.1. For any (X,S) G R x Z/2Z, i/ie restriction of to 
the Jacobi group is unitarily equivalent to the direct integral of unitary 
representations w p via Tt (see (14.21) ): 

(4.10) T^ixMcJ - / tUpdp. 
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Next we establish the link between the representations (w p , L 2 (M. 2m )) 
and L 2 (M m )) of the Heisenberg group N ~ jj^m+i _ p or we 
note that the representation w p brings us to the changeover of one 
parameter families of automorphisms of H 2m+1 , from {r p : p G 1R X } to 
{ip p : p G M x } which defined by 



(4.11) ip p (s, a, a) := -s, -a, - a 

VP 2 p 

Then we state the following covariance relation given by Op Tp : 

Lemma 4.2. For every g G H 2m+1 the following identity in End(L 2 (M m )) 
holds for any 6 G L 2 (M 2m ) : 

(4.12) Op Tp (w p (g)G) = 0p Tp (6) o^fcr 1 ). 

Proof. Let g = (s, a, a) G H 2m+1 and take an arbitrary function k G 
L 2 (M. m ). Using the integral formula f 1 3 . 1 2 j) for Op , we get 

Op Tp (rxj p (g)&)u(x) 
= [ {w p {g)&) (?±1L, -A e 2 ^-y^u{y)dydi 



m v rom 



where 

P ( \ /x + y + 2a \\ / „ . p 

B = /OS- - I (-£ + a,a) - f - , a) 1 - {x - y - 2a, £ + -a 

= P« + + - (x-y,£). 
In view of the definitions ( 13 . 13[) anf (14.111) . 

^(g- 1 ) = V^g- 1 ) = &(-ps, ~2a, - P -a). 

Thus, by the definition (13. 2 \ of the Schrodinger representation d, we 
have 

(^ p (s^)u)(y) = e~ 2 ^ s+E ^ a+ y^u(y + 2a). 
Hence, the last integral equals 

/ G (i±*, -i] (^(g-^uMe^-^dydt 
Pr (G)^ (g-^u) (x). 
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□ 

Then, it turns out that the decomposition (14.101) is not irreducible, 
but the following lemma holds: 

Lemma 4.3. For any p G 1R X , w p is a unitary representation of the 
Jacobi group G J on L 2 (R 2m ), which splits into a direct sum w° p © w\ 
of two pairwise inequivalent unitary irreducible representations . 

Proof. Consider the rescaling map r p introduced by (13.101) and recall 
that the r p -twisted Weyl quantization map induces a G J equivariant 
isomorphism 

(4.13) Op Tp : L 2 {R 2rn ) ^ HS(L 2 (M m ), L 2 (IR m )) 

intertwining the w p and actions (14. 12H . 

The irreducibility of the Schrodinger representation d p of the group 
iV (Fact 13. 3j) implies therefore that any iV-invariant closed subspace 
in HS(L 2 (M m ),L 2 (M m )) must be of the form HS(L 2 (M m ), U) for some 
closed subspace U C L 2 (R m ). 

In view of the covariance relation (I3.17P of the Weyl quantization, the 
subspace HS(L 2 (IR m ), U) is Sp(m, IR)-invariant if and only if U itself is 
Mp(m, IR)-invariant (see Proposition 13.21) . and the latter happens only 
if U is one of {0}, L 2 (M m ) even , L 2 (R m ) odd or L 2 (W m ). Thus, we have 
the following irreducible decomposition of w pi seen as a representation 
of G J on L 2 (R 2m ): 

L 2 (R 2m ) = W + ®W. 

(4.14) ^ HS(L 2 (M m ), L 2 (R m ) even ) © HS(L 2 (M m ), L 2 (R m ) odd ). 

°Prp 

From Proposition 13.21 2) we deduce that the corresponding represen- 
tations, to be denoted by zu p , of G J , where 5 labels the parity, are 
pairwise inequivalent, i.e. tu p = w s , if and only if p = p' and 5 = 5' for 
all p,p' G R and 5, 5' G Z/2Z. '' □ 

The following lemma is straightforward from the definition of the 
involution 6 i-)- 6^ (see (13. 19|) ) . 

Lemma 4.4. The subspaces W + and W- introduced above are the +1 
and —1 eigenspaces of the involution & t— >• &p, respectively. 

Eventually, we take the A-action into account, and give the branching 
law of the (degenerate) principal series representation tc^s of G\ when 
restricted to the maximal parabolic subgroup MAN . 
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Lemma 4.5. (Branching law for G\ \, MAN). For every (A, 5) G 
R x Z/2Z the space Vixj acted upon by the representation itix^MAN 
splits into the direct sum of four irreducible representations: 

(4.15) v iX , 5 ^ nl(w + ) © nl(W-) © ni(w + ) © ni(W-). 

Proof. We shall prove first that each summand in (I4.15P is already 
irreducible as a representation of M AN ~ G J A. Then we see that 
it is stable by the group MAN and thus irreducible because M is 
generated by M Q and —Ii n , which acts on V^s by the scalar (— l) s . 

In light of the G J -irreducible decomposition f)4.10p . any G J - invariant 
closed subspace U of V^a must be of the form 

U = JT 1 (L 2 (E + , W+)) © T;\L\E^ W-)), 

for some measurable sets E± in K. 

Suppose furthermore that U is A-invariant. Notice that the group 
A acts on V iX ,s ^ L 2 (R 2m+1 ) by 

ir iXj5 {a)f(t,X) = ar l - m - iX f(ar%ar l X). 

In turn, their partial Fourier transforms with respect to the t G R 
variable are given by 

(T t 7r a A a )f)(P' X ) = a l - m - l \7 t f){a 2 Pl a^X). 

Therefore, Ttf is supported in E± if and only if J r t'^i\,s(o J )f is supported 
in a~ 2 E± as a PV±-valued function on R. In particular, U is an A- 
invariant subspace if and only if E± is an invariant measurable set 
under the dilation p i-> a 2 p (a > 0), namely, E± = {0}, R_, R + , or R 
(up to measure zero sets). 

Since M AN ~ G J A, M G AA^-invariant proper closed subspaces must 
be of the form ^(L 2 ^, W £ )) with e = + or -. 

We recall from Lemma ¥Z7Z\ that the Hilbert space L 2 (M, W e ) is a sum 
of W^-valued Hardy spaces: 

(4.16) L 2 (R,W £ ) =n 2 + (W £ )®n 2 _(W £ ) ^ L 2 (R + ,W £ )®L 2 (R^,W £ ). 

Ft 

Now Lemma 14.51 has been proved. □ 

Lemma 14.51 implies that the representation 7Tix,s of G\ has at most 
four irreducible subrepresentations. The precise statement for this will 
be given in Theorem 18.31 . 
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5. Restriction of 71^,5 to a maximal compact subgroup 

As the operator calculus model L 2 (M, HS(L 2 (M m ), L 2 (IR m ))) was ap- 
propriate for studying the P-structure of vr iA 5, we use complex spherical 
harmonics for the analysis of the iT-structure of these representations. 

We retain the convention n = m + 1. Identifying the symplectic 
form u on IR 2n with the imaginary part of the Hermitian inner product 
on C n we realize the group of unitary transformations K = U(n) as a 
subgroup of G\ = Sp(n,M.). Then the group K is a maximal compact 
subgroup of G\. 

Analogously to the classical spherical harmonics on MP, consider har- 
monic polynomials on C n as follows. For a, {3 G M, let T-L a '^(C n ) denote 
the vector space of polynomials p(zo, ■ ■ ■ , z m , z , . . . , z m ) on C n which 

(1) are homogeneous of degree a in (z , . . . , z m ) and of degree (3 in 

(^Cb • • • j ^m)i 

m Q2 

(2) belong to the kernel of the differential operator — — ■—. 

i=0 1 % 

Then, 7i a ' l3 (C n ) is a finite dimensional vector space. It is non-zero 
except for the case where n = 1 and a, (3 > 1. The natural action of 
K on polynomials, 

p(z , ...,z m ,Zo,...,z m )h+ p(g~ 1 (z , ...,z m ), g- l (z , z m )) (g G K), 

leaves "H Q,/3 (C ra ) invariant. The resulting representations of K on / H a '"(C n ), 
which we denote by the same symbol n a '^(C n ), are irreducible and 
pairwise inequivalent for any such a, (3. 

The restriction of H a,l3 (C n ) to the unit sphere S 2m+1 = {(z , . . . , z m ) G 
C n : Y^jLo \ z j\ 2 = 1} * s i n J ec ti ve an d gives a complete orthogonal basis 
of L 2 (S 2m+1 ), and we have a discrete sum decomposition 

(5.1) L 2 {S 2m+1 ) ~ ^®^(C n )| S 2 m+ i (m>l). 

a,/3eN 

The case m = collapses to 

a£N /3GN+ 

Fixing a p G C we may extend functions on S 2m+1 to homogeneous 
functions of degree — (m + 1 + p). The decomposition (15. ip gives rise 
to the branching law (K-type formula) with respect to the maximal 
compact subgroup. 
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Lemma 5.1. (Branching law for G% \. K). The restriction of tc^j 
to the subgroup K of G\ is decomposed into a discrete direct sum of 
pairwise inequivalent representations : 



H a >P(C n ) (m > 1) 



a,/3eN 
a+/3=S mod 2 



ir^ s \ K ~ ^ e ^°(C)© £® n°' p (C) (m = 0). 

a£N /3eN+ 

a=5 mod 2 p=& mo d 2 

We shall refer to % ol, °(C n ) as a i^-type of the representation 7r M>( 5. 

The restriction Gi 1 K is multiplicity free. Therefore any iT-inter- 
twining operator (in particular, any Gi-intertwining operator) acts as a 
scalar on every il~-type by Schur's lemma. We give an explicit formula 
of this scalar for the Knapp-Stein intertwining operator: 

7^,(5 : V-fj, ; s — > V^s, 

which is defined as the meromorphic continuation of the following in- 
tegral operator 

(%, S f)(v) := [ /(0 Htv)\ {-^(sgnco^rifdo-iO. 
Js 2 ™- 1 

Here da is the Euclidean measure on the unit sphere. Further, we 
normalize it by 

(5 - 2) % ' s := cdb) 7 "^' 



where 

C 2n ^,8) :=2tt^5 x 



r(^) 

p/ 2-fi-n - 



(S = 0), 



~^ r ^+n+i- y (8 — 1, 



Proposition 5.2. For a, (3 G N ; we set 5 = a + (3 mod 2. The normal- 
ized Knapp-Stein intertwining operator T^s acts on 'H a,l3 (C n ) as the 
following scalar 

f-DV-^ 2 - '- 

Proof. See [HI Theorem 2.1] for 5 = 0. The proof for 5 = 1 works as 
well by using Lemma 15.41 □ 
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Remark 5.3. Without normalization, the Knapp-Stein intertwining op- 
erator T^s acts on K Q,/3 (C 1 ) as 



where 5 = a + (3 mod 2 and 



= (-if A a+0 (ii) id, 



A k (fi) := 2<K n ~* 



■p / 1 — /.t — n ^ 



2 

r (^-) -^^Mrrf (*e2N + l), 



The symplectic Fourier transform J- Symp , defined by (13.61) . may be 
written as: 

(^symp/X^) = / f{X)e- 2m ^ X ^dX = (JT R2n/ ) (JY), 

where J : IR 2n — > M. 2n is given by £) := (— £, x). 

For generic complex parameter fi (e.g. fi ^ n,n + 2, . . . for 5 = 0), 
the space V™ s of homogeneous functions on IR 2n \ {0} may be regarded 
as a subspace of the space iS'(M 2ri ) of tempered distributions, and we 
have the following commutative diagram: 

F symp : S'(R 2n ) ^ S'(R 2n ) 

u o u 

Lemma 5.4. As operators that depend meromorphically on fi, 
satisfy the following identity: 

Proof. The proof parallels that of [61 Proposition 2.3]. For h G C°°(S 2n ~ 1 )s, 
we define a homogeneous function h^ n G V™ s by 

VnK) := r»- n h(0 (r > 0, f G S 2 "- 1 ). 
Then we recall from [Bj Proposition 2.2] the following formula: 

where ((£, r])—i0) x is a distribution of £, 77, obtained by the substitution 
of t = (£, r/) into the distribution (t — i0) x of one variable t. 
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To conclude, we use 



e^vw; ( cos^^ — '-{t^ n - ism — ^|t| - ^ "sgnt 



□ 



We note that the Knapp-Stein intertwining operator induces a uni- 
tary equivalence of representations and 7r_jA,5 of Gi = Sp(n, M): 

(5.3) 7TjA,5 — 7T-iA,5 ; for any A G IR and 5 G Z/2Z. 

6. Algebraic Knapp-Stein intertwining operator 

We introduce yet another model W Mi <5 — L 2 (IR 2m+1 ), referred to as 
the non-standard model, of the representation ir^s as the image of the 
partial Fourier transform 

: L 2 (R 1+m+m ) L 2 (M 1+m+m ), 

where £ denotes the last variable in W 71 . Then the space U^s inherits a 
Gi-module structure from (tt^s, V/ij) through o T t (see Figure [47T]) . 

The advantage of this model is that the Knapp-Stein intertwining 
operator becomes an algebraic operator (see Theorem 16.11 below) . The 
price to pay is that the Lie algebra t acts on U^s by second order 
differential operators. We can still give an explicit form of minimal K- 
types on the model U^s when it splits into two irreducible components 
(/x = 0, 5 = 0, 1) by means of il~-Bessel functions (SectiorGj). 

We define an endomorphism of L 2 (IR 2m+1 ) by 

(6.1) {T. s H)(p,x, V ) := £ '"(sgnpYH ( p, \ P -x 



2 



P"2 



Regarding T^s as an operator on the iV-picture, we have 

Theorem 6.1 (algebraic Knapp-Stein intertwining operator). For any 

p G C and 5 G Z/2Z ; the following diagram commutes: 



H,5 *~ ^11,5 



IA- „ x > IA, 
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To prove Theorem 16. 11 we work on the ambient space IR 2n (= IR 2m+2 ). 
Let J-jgn denote the partial Fourier transform of the last n coordinates 
in R 2n . 

Lemma 6.2. 1) For f G V-^g, the function J-m n f satisfies 

(J~M. n f) r ^jf) = \r f (sgn r) s '(J^ n f)(x,T]), r G R x , x,7] G R n . 

2) For f G S'(R 2n ),x,r] G R n ; we /iawe 

(J=k» o J" symp o J^n)f(x, = /(£, ar). 

Proof of Lemma \6. 2\ 1) This is a straightforward computation. 
2)For/(x,£')eS(M 2 "), 

(J* R » o j; ymp o T^nf)(y, rf) 

/(x,^)e 27ri< ^^' > e" 27ri( ^' 2/> ~ <a: '' 7>) e- 27ri( ' 7 '' 7 ' > ^ / dx^(i7; 



« ./KM™ ./IB" 



f{x,Z')8{£' -y)5{rf -x)dxdg 

= f(v',y)- 

□ 

From now x,£,,r) will stand again for elements of IR m , where m = 
n — 1. 

Proof of Theorem \6.1\ According to the choice of the isomorphism (14.41) 
between the Lie group N and the standard Heisenberg Lie group, for 
/ G V-^s, we set 

F(t,x,£) := /(l,x,2t,0, 
H(p,x,r}) := (T t ^F)(p,x,ri), 

where t,p G E and z,f G M m . Then H(p,x,rj) = 5pTi»/)(l,aJ, §,77). 
Thus, according to Lemma 16.21 

J r 4 J r € (J r symp /)(p,x,r7) = ij r R n(J r symp /)(l,x, ^,77) 



2 p p 
p V, 2 , 2 



sgnp) 5 (.F RB /)(l, -77, -, -x) 



^ (sgnp) 5 #(p, -77, £ar). 
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Now Theorem follows from Lemma [5.41 □ 

7. Minimal J^-type in a non-standard model 

We give an explicit formula for two particular fT-fmite vectors of 
tcq : s (in fact, minimal i^-types of irreducible components 7t os of 710,5; 
see Theorem 18.31 1)) in the non-standard L 2 -model Wo,s(~ L 2 (IR 2m+1 )). 
The main results (see Proposition 17.11) show that minimal i^-types are 
represented in terms of f^-Bessel functions in this model. Although we 
do not use these results in the proof of Theorem 18.31 we think they are 
interesting of their own from the view point of geometric analysis of 
small representations. It is noteworthy that similar feature to Proposi- 
tion [7j] has been observed in the L 2 -model of minimal representations 
of some other reductive groups (see e.g. [HI Part III]). 

We begin with the identification 

C ^ H°'°(C m+1 ), 1 h+ 1 (constant function), 

and extend it to a homogeneous function on IR 2n belonging to V^o (see 
( 14. ip ). Using the formula ( 14. 5 p in the iV-picture, we set 

h + (t,x,£) := (1 + At 2 + \x\ 2 + \^)-^ . 

Notice that h + (t,x,£) G V 0fi n %°'°(C m+1 ) in the K-type formula of 
7T 0i o (see Lemma EH]). 
Let 

(7.1) 1>(p,x,v):=(l + \x\ 2 )*(£ + \v\ 2 )*. 

Likewise we identify 

m 

C m+1 J- ^0,l( C m+l) ; b ^ J^bjZj, 

3=0 

and set 
(7.2) 

m 

hi(t, x,0 := (1 + 4t 2 + \x\ 2 + \^\ 2 r^(b (l - 2it) +Y, b j( x i ~ 

3=1 

(7.3) Vb ( P ,x, V ) : =^(( ( ( 4+'h4) ' 6 ° (§) 

where u denotes the standard symplectic form on C 2 defined as in (13.11) . 
Then G V 1 n"H 0,1 (C m+1 ) in the .fT-type formula of 7T 1 (see Lemma 

E3J. 
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Let K u (z) denote the modified Bessel function of the second kind 
(i^-Bessel function for short). Then the K- finite vectors h + and 
[b G C m+1 ) in the standard model (iV-picture) are of the following 
form in the non-standard model Uq 5. 



Proposition 7.1 

/,.'. .r. it ) ■ ^ 



m + 2 

7T 2 



1) (T t ^h + )(p,x,r]) = rfm+1 . K (2mp(p,x,ri)). 



2 I 

m+2 , ^ 

7T 2 (p b (p,X,Tj) 



2) (F t F<:h b )(p,x,r}) = 2r (m+2) ^p X \) ex P (-^(P^^))- 

The rest of this section is devoted to the proof of Proposition 17. II In 
order to get simpler formulas we also use the following normalization 
K v {z) := (§)"" ' K v {z) [201 Section 7.2]. 

Lemma 7.2. For every p & M. let us define the following function on 
RxR m : 

I, = 1^,7]):= [ (a 2 + Itfye-^M- 

Then, 

rn 

(7.4) I,(a, V ) = -^a m - 2 »K f ^(2na\ V \). 

Proof. Recall the classical Bochner formula 

f e- 2i7rs ^da(£) = 2irs 1 -fj f „ 1 (2irs), for£' G S" 11-1 , 
Js™- 1 

where J v {z) denotes the Bessel function of the first kind. Then, 



I^ri) = / / (a 2 + r 2 )-^e- 2inrlr,l{ ^ ) r m - 1 drda(0 
Jo Js™- 1 

/"OO 

= 2ir\i]\ 1 ~^ / J™_i(2nr\r]\)(r 2 + a 2 y^dr. 
Jo 

According to [H 8.5 (20)] we have 

/ x 2 + a 2 )- f "- 1 JJxy)(xy)^dx = y —- — f . 

7 2T(// + 1) 

for Re a > and — 1 < Re z/ < 2 Re /i + |, which implies 

27r M / a \ 



J " 2 -a m - 2 ^_ M (2vra|r/|). 
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□ 



In particular, we have 

tt^ exp(-27ra|r/|) 

l m+l a, 77 = — ; — rn , 

— V > /; r (I2±l) a 

27T - It/I 

iWa,??) = , m+2 s — ffi(27raM). 
1 (,-2-J a 

Here we used K_i(z) = -^e~ z in the first identity. By a little abuse of 
notation, we write hZ-. and hZ-. for hZ Q Q s and hZ 1 respectively 

Lemma 7.3. For G R x IR m ; we set 

a = a{t,x) : = 0. + 4i 2 + \x\ 2 . 

Then, 

x, 77) = Im+i(a(t,x),r}), 

Proof. By definition 

(J- € /i + )(t,x,r / )= / (l+4t 2 + |x| 2 + |er)-^e- 2 ^^ 

= / (l + 4t 2 + |x| 2 + |er)-^(x 1 -^i)e- 27r ^ > ^ 

i a 



( I1 + ^) / =* 2 (^ 1 + 4(2 + wv ")- 



27r drji . 

Hence Lemma 17.31 is proved. □ 
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Proof of Proposition \7l\ We recall from [HI vol. I, 1.4 (27); 1.13 (45); 



2.13 (43)] the following formulas: For Red > 0, Rec > and s > 0, 

r° e Xp (-^(t 2 + c 2 )2) cos(st)rft = (c(g2 d2) |, 

Jo (t 2 + C 2 )5 



(t 2 + c 2 )^ ' " ~ V 2 d"c"-5(s 2 + d 2 )W" 
2"" 1 v^F d"" c 1 - 2 " K_ u+1 (c(s 2 + d 2 ) I ) . 



tK x {d{t 2 + c 2 )a) .... vrsexp(-c(s 2 + 



2^ 



Jo (t 2 + c 2 )5 2d ( s 2 + c p)2 

We apply the formulas (17. 5p and (17. 6p with d = 47r|^|, c = |(l + |x| 2 )2 
and s = 2irp. In view that a = a(t, x) = 2(t 2 + c 2 )^ and 27T0(p, x, 77) = 
c(s 2 + (i 2 )2 , we get 

J-00 a 

1°° ^e-'* = — 1— T exp(-2^(p, *,„))■ 
i-00 a 4|r/|(l + |x| 2 )5 

Here, we have used again K_i{z) = ^ e~ z for the second equation. 
Thus the first statement has been proved. 

To see the second statement, it is sufficient to treat the following two 
cases: b = (1, 0, . . . , 0) and b = (0, 1, 0, . . . , 0). We use 



(1 d 
(xi + — — J Im±2(a(t,X),7)) 

1 d 



Now use 



J_ d \exp(-2nij(p,x,i])) _ <p(i)(p, x, rj) exp(-2vn/>(p, x, rj)) 

^1 1 ^ 



2tt<9?7i/ (l + |a;| 2 )5 ip(p,x,r]) 
The case b = (1, 0, . . . , 0) goes similarly by using the formula (17. 7p . □ 
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8. Branching law for GL(2n, E) j. Sp(n,M) 

From now we give a proof of Theorem 11.11 with emphasis on geometric 
analysis involved. 

Our strategy is the following. Suppose P is a closed subgroup of a 
Lie group G, x '■ P — > C x a unitary character, and C := G x P x a G- 
equivariant line bundle over G/P. We write L 2 (G/P, C) for the Hilbert 
space consisting of L 2 -sections for the line bundle £(g> (A top T* (G / P))z . 
Then the group G acts on L 2 (G/P,C) as a unitary representation, to 
be denoted by tt^, by translations. 

If (G, H) is a reductive symmetric pair and P is a parabolic subgroup 
of G, then there exist finitely many open iJ-orbits on the real flag 
variety G/P such that UjO^ is open dense in G/P. (In our cases 
below, the number of open if -orbits is at most two.) Applying the 
Mackey theory, we see that the restriction of the unitary representation 
tt^ to the subgroup H is unitarily equivalent to a finite direct sum: 

4\b^®L\0^\L\ oU) ). 

3 

Thus the branching problem is reduced to the irreducible decompo- 
sition of L 2 (0^\ C\ u)), equivalently, the Plancherel formula for the 
homogeneous line bundle C\ <j) over open #-orbits O u) . 

In our specific setting, where G = GL(N, M.) and P = Pn (see 
(11. 2p ). the base space G/P is the real projective space P^ -1 ]^. For 
(A, 5) 6 R X Z/2Z, we define a unitary character Xi\,6 of Pn by 

a| A (sgna)' 5 , a e GL(1,M), C G GL(N -1,M), b G R N ' 

in the matrix realization of Pn- Then ti2 coincides with 7rjR x in 
previous notation. In this and the next three sections, we find the 
explicit irreducible decomposition of L 2 (O^\C\ O 0)) with respect to 

We begin with the case H = Gi, i.e. 

(G, H) = (GL(2n, E), 5p(n,R)). 

As we have already seen in Section 0] the group Gi acts transitively 
on G/Pn, and we have the following unitary equivalence of unitary 
representations of G\ = Sp(n,R): 



*6 
C 
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Here vr iA p ^ n ' is a unitary representation of Sp(n, R) induced from the 
maximal parabolic subgroup P = GiflP/v — (GL(1, R) x Spin — 1, R)) x 

Thus the following two statements are equivalent. 

Theorem 8.1. The restriction of V^^ 2 " /rom GL(2n } R) to Sp(n, R) 
stays irreducible for any A G R x and 5 G {0,1}. It splits into two 
irreducible components for A = 0, 5 = 0, 1 and n > 2. 

Theorem 8.2. Let P be a maximal parabolic subgroup of Gi whose 
Levi part is isomorphic to GL(1,R) x Sp(n — 1,R), and denote by 
T^ix,5 (X &M.,5 — 0,1) the corresponding unitary (degenerate) principal 
series representation of G\. Then for n > 2, ~Ki\j are irreducible for 
any (A, 5) G M x x Z/2Z, and splits into a direct sum of two irreducible 
components for A = 0, 5 = 0, 1. 

Theorem 18.21 itself was proved in [211 Theorem 7.3]. The case of 
5 = was studied by different methods earlier in [TU] and also very 
recently in [2] (A = and 6 = 0) in the context of special unipotent 
representations of the split group Sp(n,M). We give yet another proof 
of Theorem 18.21 in the most interesting case, i.e. in the case A = and 
5 = 0, 1 below. 

Theorem 18.31 describes a finer structure of the irreducible summands. 
The novelty here (even for the 5 = case) is that we characterize 
explicitly the two irreducible summands by their i\"-module structure, 
and also by their P-module structure. The former is given in terms of 
complex spherical harmonics ( cf. Lemma 15 . 1 j) and the latter in terms 
of Hardy spaces (cf. Lemma [4. 5 p . as follows: 

Theorem 8.3. Let n > 2 and 5 G Z/2Z. The unitary representa- 
tion 7r 0> 5 of G% = Sp(n, R) splits into the direct sum of two irreducible 
representations of ' G\: 

(8.1) vr ,5 = 7T+ 5 © ir~ s . 

1) (Characterization by K-type). Each irreducible summand in \8.1\) 
has the following K-type formula: 

/3e2N 
a=0+S mod 2 

/3G2N+1 
a=0+S mod 2 

where ^® denotes the Hilbert completion of the algebraic direct sum. 
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2) (Characterization by Hardy spaces). The irreducible summands 
tIq 5 consist of two Hardy spaces via the isomorphism J^. 15\ ): 



7T+ ~ Hi(w+) © Ht(w+), 7r - ~ n%{wj) © nt(w_ 
4,i 



71+, ~ n 2 + (w+) © Hl(W-), TT Q1 ~ ft 2 (WL) © ^ 2 _(^ + 



i/ere, W± are i/ie subspaces of L 2 (R 2m ) defined in (14.141) . and W±(W £ ) 
are the W e -valued Hardy spaces. 

3) ( Characterization by the Knapp-Stein intertwining operator). The 
irreducible summands ir^ s are the ±1 eigenspaces of the normalized 

Knapp-Stein intertwining operator 7o,<5 (see j[5.2\) ). 

Proof. 1) and 3) The normalized Knapp-Stein intertwining operator 
7o,(5 has eigenvalues either 1 or —1 according to the parity of the K- 
type 'H a 'P(C n ), namely (3 = or (3 = 1 mod 2 by Proposition 15.21 
Hence the statements 1) and 3) are proved. 

2) In the model U ,s ^ L 2 (R 2m+l ) (see Section ED, the Knapp-Stein 
intertwining operator 7o,<5 is equivalent to the algebraic operator 

T ,s : H(p,x,r}) ->• (sgn p) s H (p, -77, 

by Theorem 16.11 

In turn, it follows from Lemma 13.41 that Tq s is transfered to the 
operator 

(8.2) 60,*)^ (sgn p) 5 &^(p,*) 

in the operator calculus model L 2 (R, HS(L 2 (M m ), L 2 (R m ))) (see Figure 
14. ip . In view of the ±1 eigenspaces of the transform (18.21) . we see that 
the statement 2) follows from the characterization of W± (see Lemma 
14. 4p and the isomorphism T% '■ 7i±(W e ) — > L 2 (1L|-, W £ ) given in Lemma 

o 

Finally, we need to prove that the summands 71"^ are irreducible Gi- 
modules. This is deduced from the decomposition of 7Tq S by means of 
Hardy spaces in 2) and from the following lemma. □ 

Lemma 8.4. For any 5 G Z/2Z, none of the Hardy spaces 'H\{W £ ) 
(e = ±) is G\-stable with respect to 7r 0>( $. 
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Proof. For Z := (z h . . . , z m ) = x + i£ G C m ~ R 2m (see (IQjl ). we set 

/«„,(«, x,0 :=(l + 4t 2 + |x| 2 + |e| 2 )-^, 

/ 0il (t, x, := (1 + 4t 2 + |x| 2 + lei 2 )" 2 ^ (X! - 

/i, (i, x, := (1 + 4t 2 + |x| 2 + Kl 2 )- 2 ^ (xi + 

/ 14 (t, x, := (1 + 4t 2 + |x| 2 + lel 2 )-^ (1 + 4t 2 - x\ - e 2 ). 

We note that /o,o = h + and /o,i = h7 1 % = in the notation of 
Section [3 Then we have f a ,/3 G % a,/3 (C n ) for any a, /3 G {0, 1}. In view 
of Theorem 18.31 1), we get 

/„ il (t,^)e#(C' , )c^, 

/ 1 , 1 (t, I) ()6K 1 ' 1 (C n )c^ 

where V^ § stands for the representaion space in the iV-picture corre- 
sponding to 7Tq ^ in Theorem 18.31 Suppose now that one of the Hardy 
spaces ?i±(W E ) were Gi-stable with respect to ttq^. Then its orthogo- 
nal complementary subspace for the decomposition in Theorem 18.31 2) 
would be also G^-stable. Since K-type is multiplicity-free in 7r 0;< 5 by 
Lemma loTTj either 7t±(W e ) or its complementary subspace should con- 
tain the .fT-type % Q ' /3 (C ri ) for some a, j3 = or 1. But this never hap- 
pens because f a ^(t,x,^) = f a> p(—t,x,£) and thus supp J-tfa,p ^ R± 
(see Lemma [2.21 4)). Thus lemma is proved. □ 

Remark 8.5. The case n — 1 is well known. Here the group 

is isomorphic to SL(2,M), and tt^s are irreducible except for (A, 5) = 

(0,1), while 710,1 splits into the direct sum of two irreducible unitary 

representations: 

a£2N+l / \ /3S2N+1 

The spaces 'H a,0 (C) and 7{ 0,/3 (C) are one dimensional, and 

(t + i) a (t 2 + g ^ Q '°(C) n v ,i, 
(t-^ + ir^G^Qnv 
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The former function extends holomorphically to the upper half plane 
n + , and the latter one extends holomorphically to n_ if a, (3 = 1 mod 
2, namely, if S = 1. 

As formulated in Theorem l8.2[ our result may be compared with gen- 
eral theory on (degenerate) principal series representations of real re- 
ductive groups. For instance, according to Harish- Chandra and Vogan- 
Wallach [25], such representations are at most a finite sum of irreducible 
representations and are 'generically' irreducible. A theorem of Kostant 
[22] asserts that spherical unitary principal series representations (in- 
duced from minimal parabolic subgroups) are irreducible. 

There has been also extensive research on the structure of (degener- 
ate) principal series representations in specific cases, in particular, in 
the case where the unipotent radical of P is abelian by A. U. Klimyk, B. 
Gruber, R. Howe, E.-T. Tan, S.-T. Lee, S. Sahi and others by algebraic 
and combinatorial methods (see e.g. [2] and references therein). 

We have not adopted here the aforementioned methods, but have 
used the idea of branching laws to non- compact subgroups (see [T7] ) 
primarily because of the belief that the latter approach to very small 
representations will open new aspects of the theory of geometric anal- 
ysis. 



Let P% = L^N^ be the standard maximal parabolic subgroup of 
GL(n, C) corresponding to the partition n = 1 + (n — 1), namely, the 
Levi subgroup of P„ is isomorphic to GL(1, C) x GL{n — 1, C) and 
the unipotent radical is the complex abelian group C n_1 . Induc- 
ing from a unitary character (u, m) G R x Z of the first factor of L^, 
GL(1, C) ~ 1R + x S 1 we define a degenerate principal series representa- 
tion tt^^"' of GL(n,C). They are pairwise inequivalent, irreducible 
unitary representations of GL(n,C) (see [HI Corollary 2.4.3]). 
We identify C n with IR 2n , and regard 

G 2 := GL(n,C) 
as a subgroup of G = GL(2n, E). 

Theorem 9.1 (Branching law GL(2n,R) I GL(n,C))- 



me2Z+<5 

Proof. The group G 2 = GL(n, C) acts transitively on the real projective 
space P 2n_1 ]R, and the unique (open) orbit O2 '■= P 2n_1 lR is represented 



9. Branching law for GL(2n,R) I GL(n,C) 



(9.1) 
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as a homogeneous space G2/H2 where the isotropy group H 2 is of the 
form 

H 2 ~ (0(1) x GL{n - 1, C))iV£. 
Since P^/H 2 — S' 1 /{±1}, we have a GVequivariant fibration: 

S7{±1} -> P 2 ™^ -> GL(n,C)/P%- 

Further, if we denote by Ca the one- dimensional representation of H 2 
obtained as the following compositions: 

H 2 H 2 /GL(n - 1, C)JVj A C x , 

then the G-equivariant line bundle £jA,<5 = G x p Ci\ t $ is represented as 
a GVequivariant line bundle simply by 

C s := Cix,s\o 2 ^ GL{n,C) Xh 2 C s . 
Therefore, we have an isomorphism as unitary representations of G 2 : 

1 ,GL{2nX)\ ^ j2 (n p \ 
H i\,S \G 2 — L [ U 2, C S )■ 

Taking the Fourier series expansion of L 2 (0 2 , Cs) along the fiber S 1 / {±1}, 
we get the irreducible decomposition (19.1 1) . □ 

An interesting feature of Theorem [9TT] is that the degenerate principal 
series representation ftf^} 2 ™'^ is discretely decomposable with respect 
to the restriction GL(2n, R) \. GL(n,C). We have seen this by find- 
ing explicit branching law, however, discrete decomposability of the 
restriction vr^ 2n ' R ^| Gi („ iC ) can be explained also by the general theory 
[16] as follows: 

Let t be a Cartan subalgebra of o(2n), and we take a standard ba- 
sis . . . , f n } in it* such that the dominant Weyl chamber for the 
disconnected group K = 0(2n) is given as 

it\ = {(A 1; . . . , A n ) : Ax > A 2 > • • • > A n > 0}. 

For K 2 := G 2 C\K ~ U (n) the Hamiltonian action of K on the cotangent 
bundle T*(K/K 2 ) has the momentum map T*(K/K 2 ) — > it*. The 
intersection of its image with the dominant Weyl chamber it* + is given 
by 

it* + nAd v (if)(^) 

= {(Ai, . . . , A n ) G it* + : A 2 i-i = Aj for 1 < % < 

On the other hand, it follows from Lemma 15.11 that the asymptotic 
^-support of iiix t s amounts to 

AS K (ira,s) = R+(l, 0, . . . , 0). 
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Hence, the triple (G, G 2 , iri\ t s) satisfies 

(9.2) ASk^s) n Ad v (K)(i^) = {0}. 

This is nothing but the criterion for discrete decomposability of the 
restriction of the unitary representation 71^,5 |g 2 (PS Theorem 2,9]). 
For Gi = Sp(n,R), we saw in Theorem 18.11 that the restriction 

7r iA^ 2n,R '' I Gi stays irreducible. Thus, this is another (obvious) exam- 
ple of discretely decomposable branching law. We can see this fact 
directly from the observation that G\ and G2 have the same maximal 
compact subgroups, 

(Ji~i :=)K n G x = K n G 2 {=: K 2 ). 

In fact, we get from f !9.2j) 

^(7r iA , 5 )nAd v (^)(^) = {0}. 

Therefore, the restriction 71^,5 Id is discretely decomposable, too. 

Remark 9.2. In contrast to the restriction of the quantization of elliptic 
orbits (equivalently, of Zuckerman's A q (A)-modules), it is rare that the 
restriction of the quantization of hyperbolic orbits (equivalently, unitar- 
ily induced representations from real parabolic subgroups) is discretely 
decomposable with respect to non-compact reductive subgroups. An- 
other discretely decomposable case was found by Lee-Loke in their 
study of the Jordan-Holder series of a certain degenerate principal se- 
ries representations. 



10. Branching law for GL(N,R) | GL(p,R) x GL(q,R) 



Let N = p + q (p, q > 1), and consider a subgroup G3 := GL(p, R) x 
GL(q,R) in G := GL(N,R). The restriction of ir?^ Nfi) with respect 
to the symmetric pair 

(G,G 3 ) = (GL(N,R),GL(p,R) x GL(q,M)) 

is decomposed into the same family of degenerate principal series rep- 
resentations of G f. 

Theorem 10.1 (Branching law GL(p + q,R) | GL(p,R) x GL(q, R)). 

GL(p+q,R), ^ / G?L(p,R) M GL(q,R) »/ 

i\,8 lG 3 - / ^A'.i' ^ i(\—\'),5—5' ■ 

<5'=0,1 ^ R 



GEOMETRIC ANALYSIS ON SMALL REPRESENTATIONS OF GL(N, R) 37 



Outline of Proof. The proof is similar to that of Theorem 19.11 The 
group G 3 = GL(p,R) x GL(q,R) acts on P p+9-1 IR with an open dense 
orbit O3 which has a GVequivariant fibration 

R x -> O, (GL(p,R)/P p ) x (GL(q,R)/P q ) . 

Hence, taking the Mellin transform by the R x -action along the fiber, 
we get Theorem 110.11 □ 



11. Branching law for GL(N,R) I 0(p,q) 
For N = p + q, we introduce the standard quadratic form of signature 

(p, q) by 

Q(x) :=xl + --- + x 2 p - x 2 p+1 x 2 p+q for x G R p+q . 

Let G 4 be the indefinite orthogonal group defined by 

0(p, q) := {9 E GL(N, R) : Q(gx) = Q(x) for any x G R p+q }. 

For q = 0, G 4 is nothing but a maximal compact subgroup K = O(N) 
of G, and the branching law nf^^'^ |g 4 is so called the K-type formula. 

In order to describe the branching law G I G4 for general p and q, 
we introduce a family of irreducible unitary representations of G 4 , to 
be denoted by 71+^'^ (v G A+(p, g) below), 7r_^ (z/ G and 
n ivs^ ( u e ^) as follows. Let t be a compact Cartan subalgebra of $j 4 , 
and we take a standard dual basis {e^} of t such that the set of roots 
for £4 := o(p) © o(g) is given by 

A(! 4 ,t 4 ) = {±(e t ± e ,):l<*<J<|] or |] + 1 < i < 3 < [|] + [|]} 

U{± e ,:l<2<[|]}(p:odd) 

U{±ei:[|] + l<<<[|] + [§]}(?: odd). 

Then, attached to the coadjoint orbits Ad v (G 4 )(z/ej) for v G 5) 
and Ad v (G 4 )(^e[|] +1 ) for v G A + (g,p), we can define unitary repre- 
sentations of G 4 , to be denoted by 7r+^ and tt^' 9 ^ as their geo- 
metric quantizations. These representations are realized in Dolbeault 
cohomologies over the corresponding coadjoint orbits endowed with G4- 
invariant complex structures, and their underlying (gc, i^)-modules are 
obtained also as cohomologically induced representations from charac- 
ters of certain ^-stable parabolic subalgebras (see [191 §5] for details). 
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We normalize 7r+^' 9 ' 1 such that its infinitesimal character is given by 

/ P + 1 o P + Q o P + Q \P + Qn 
(", — -2, — -3,..., — -[—]) 

in the Harish-Chandra parametrization. The parameter set that we 
need for 7r+^'^ is A + (p, q) := A° + (p, q) U A\(p, q) where 

' {v G 2Z + 2^ + i + s : v > 0}, (p > 1, q ^ 0); 
{z/G2Z+£^ + l + 5: u> 2-1}, (p>l,g = 0); 
<(p,5):=«i 0, " " (p=l, (<?,*) ^(0,1)) 

or (p = 0); 

. {|}, (p = 1,(9, = (0,1))- 

Notice that the identification 0(p, q) ~ 0(q,p) induces the equivalence 

o(p,q) _ 0(,,p) 

For p, q > the group G4 = 0(p,q) is non-compact ant there are 
continuously many hyperbolic coadjoint orbits. Attached to (minimal) 
hyperbolic coadjoint orbits, we can define another family of irreducible 
unitary representations of G4, to be denoted by T^fjg' 9 ^ for v G R and 

5 G {0, 1}. Namely, let nf^' 9 ^ be the unitary representation of G 4 in- 
duced from a unitary character (iz/, 8) of a maximal parabolic subgroup 
of G 4 whose Levi part is 0(1, 1) x 0(p — 1, q — 1). 

We note that the Knapp-Stein intertwining operator gives a unitary 
isomorphism 

< ( r^^& 9) (^i, 5=0,1). 

Theorem 11.1 (Branching law GL(p + q,M) j. 0(p,q)). 



Notice that in case when q = the latter two components of the 
above decomposition do not occur and one gets the K-type formula 
GL(n,M) I 0(n). 

As a preparation of the proof, we formalize the Plancherel formula 
on the hyperboloid from a modern viewpoint of representation theory. 

Let X(p, q)± be a hypersurface in W +q defined by 

X(p,q)± := {x = (x',x") G R p+q : \x'\ 2 - \x"\ 2 = ±1}. 

We endow X(p,q)± with pseudo-Riemannian structures by restrict- 
ing ds 2 = dx\ + • • • + dx 2 — dx 2 +l — ■ ■ ■ — dx 2 +q on W J+q . Then, X(p, q)± 
becomes a space form of pseudo-Riemannian manifolds in the sense 
that its sectional curvature k is constant. To be explicit, X(p,q) + has 
a pseudo-Riemannian structure of signature (p — l,q) with sectional 
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curvature k = 1, whereas X(p, q)_ has a signature (p, q — 1) with 
k = —1. Clearly, G 4 acts on X(p, q)± as isometries. 

We denote by L 2 (X(p, q)±) the Hilbert space consisting of square 
integrable functions on X(p, q)± with respect to the induced measure 
from ds 2 \ X (p,q)- 

The irreducible decomposition of the unitary representation of G4 
on L 2 (X(p, q)±) is equivalent to the spectral decomposition of the 
Laplace-Beltrami operator on X(p, q)± with respect to the G4- invariant 
pseudo-Riemannian structures. The latter viewpoint was established 
by Faraut [H] and Strichartz [21]. 

As we saw in [19, §5], the discrete series representations on hyper- 
boloids X(p, q)± are isomorphic to ^ < ±f, ,q " > with parameter set A±(p, q). 

(11.1) L 2 (X(p,q) + ) s 

(11.2) L 2 (X(p,q)_) s 

Here we note that each irreducible decomposition is multiplicity free, 
the continuous spectra in both decompositions are the same and the 
discrete ones are distinct. 

Proof of Theorem \11.1\ According to the decomposition 

R p+q D {x e R p+q : Q{x) > 0} U {x e R p+q : Q{x) < 0}, 

dense 

the group G 4 = 0(p, q) acts on P J,+9 ~ 1 IR with two open orbits, denoted 
by 0\ and 0~l . A distinguishing feature for G 4 is that these open 
G 4 -orbits are reductive homogeneous spaces. To be explicit, let H± 
and H± be the isotropy subgroups of G4 at [ei] G 0\ and [e p+q ] G Cj, 
respectively, where {e^} denotes the standard basis of W +q . Then we 
have 

0\ * G^l Ht = 0(p, q)/(0(l) x 0(p - 1, q)), 
O4 * G^jEl = 0(p, q)/(0(p, q-l)x 0(1)). 

Correspondingly, the restriction of the line bundle Ci\ t s = G x P x%\,s 
to the open sets of the base space G/P is given by 

G4 x H ± Cs, 



E - 

u&A s + (q,p) 



0{p,q) 



G{p,q)i 
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where C$ is a one- dimensional representation of Hf defined by 

0(1) x 0(p-l,q) -> C x , (a, A) ^ a 5 , 

0(p,9-l)xO(l)^C x , {B,b)^b 5 , 

respectively. It is noteworthy that unlike the cases G2 = GL(n, C) and 
G3 = GL(p, R) x GL(q, R), the continuous parameter A is not involved 
in dm]). 

Since the union 0\ U O J is open dense in P P+(?_1 R, we have a G4- 
unitary equivalence (independent of A): 

Sections for the line bundle G4 x H ± Cg over are identified with 
even functions (5 = 0) or odd functions (5 = 1) on hyperboloids 
X(p, q)± because X(p, q)± are double covering manifolds of Of. 

According to the parity of functions on the hyperboloid X(p, q)±, we 
decompose 

L 2 (X(p, q)±) = L 2 (X(p, q)±) © L 2 (X(p, q) ± ) v 
Hence, we get Theorem 111.11 □ 

12. Tensor products Met v © Met 



The irreducible decomposition of the tensor product of two represen- 
tations is a special example of branching laws. It is well-understood 
that the tensor product of the same Segal-Shale-Weil representation 
(e.g. Met ©Met) decomposes into a discrete direct sum of lowest weight 
representations of Sp(n, R) (see [U]). In this section, we prove: 

Theorem 12.1. Let Met be the Segal-Shale-Weil representation of 
the metaplectic group Mp(n,M), and Met v its contragredient represen- 
tation. Then the tensor product representation Met v © Met is well- 
defined as a representation of Sp(n, R), and decomposes into the direct 
integral of irreducible unitary representations as follows: 

(12.1) Met v ©Met~^ T 2vrf A p J n ' R) dA. 

5=0,1 

Remark 12.2. The branching formula in Theorem 112.11 may be re- 
garded as the dual pair correspondence 0(1, 1) • Sp(n, R) with respect 
to the Segal-Shale- Weil representation of Mp(2n, R). We note that the 
Lie group 0(1, 1) is non-abelian, and its finite dimensional irreducible 
unitary representations are generically of dimension two, which corre- 
sponds the multiplicity two in the right-hand side of (112. ip . 
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Proof. By Fact 13.21 the Weyl operatorcalculus 

(12.2) Op : L 2 (R 2n ) ^ HS(L 2 (M n ), L 2 (R n )) 

gives an intertwining operator as unitary representations of Mp(n,R). 
We write L 2 (M n ) v for the dual Hilbert space, and identify 

(12.3) HS(L 2 (M n ), L 2 {R n )) ~ L 2 (R n f®L 2 (R n ), 

where <g) denotes the completion of the tensor product of Hilbert spaces. 
Composing ( 112. 2p and ( I12.3p . we see that the tensor product represen- 
tation Met v eg) Met of Mp(n, R) is unitarily equivalent to the regular 
representation on L 2 (IR 2n ). This representation on the phase space 
L (R ) is well-defined representation of Sp(n, R). 

We consider the Mellin transform on IR 2n , which is defined as the 
Fourier transform along the radial direction: 

1 r°° 

f^^J \t\ n - 1+l \sgnt) 5 f(tX)dt, 

with A G R, 5 = 0, 1,X G R 2n . Then, the Mellin transform gives a 
spectral decomposition of the Hilbert space L 2 (IR 2n ). Therefore, the 
phase space representation L 2 (IR 2n ) is decomposed as a direct integral 
of Hilbert spaces: 

(12.4) L 2 (R 2n )^ J2 / V ^sdX. 



8=0.1 



Since vr^ n ' M ' ) ~ vrf^ MJ (see (I5.3P ). we get Theorem 112.11 □ 
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